Image capture position and image capture direction estimation device, image capture device, image capture position and image capture direction estimation method and program

ABSTRACT

An image capture position and direction estimation device includes a region determination unit that determines a plurality of regions to be associated between a query image and an image with position and direction, a depth estimation unit that estimates a depth of each region, and an image capture position and direction estimation unit that estimates a direction in the region with a large depth estimated by the depth estimation unit and estimates a position in the region with a small depth estimated by the depth estimation unit.

TECHNICAL FIELD

The present invention relates to an image capture position and imagecapture direction estimation device (image capture position directionestimation device), an image capture device, an image capture positionand image capture direction estimation method (image capture positiondirection estimation method) and a program, and more particularly, animage capture position and image capture direction estimation device, animage capture device, an image capture position and image capturedirection estimation method and a program in which landmark positioninginformation is not necessary.

BACKGROUND ART

As methods of estimating a self-position at the time of image capturebased on an image captured by a camera, methods (Patent Documents 1 and2) of estimating a self-position from a plurality of landmarks detectedfrom an image and a method (Non-Patent Document 1) of performingmatching with a previously captured image group to which a position anda direction are assigned have been suggested. Here, the self-positionrefers to information indicating a position and an image capturedirection at the time of image capture.

Patent Document 1 discloses a technology for estimating a self-positionbased on position information, in which a building, a mark, or the likeused as a landmark is positioned in advance by a highly accurate GPS(Global Positioning System) device, and a camera parameter of an imagecapture device.

In the estimation method, an angle is calculated per pixel from a cameraparameter of the camera used to capture an image, a constrained circlein which there is a probability that the self-position is present isdefined from the maximum angle of nip and the minimum angle of nipbetween landmarks, and a score is calculated using a model which definesa likelihood of an angle of nip. Further, the score calculated for eachof the spaces between the landmarks is added to a coordinate positionand the coordinate position with the maximum score is estimated as theself-position.

Patent Document 2 discloses a method of estimating a self-position inconsideration of the degree of coincidence when landmarks are detected.In the estimation method, the degree of coincidence between an object ina captured image and a landmark is calculated using a database thatstores the positions of landmarks on a map and an amount of acharacteristic. Thereafter, a self-existence range used to estimate theself-position in the order of the higher degree of coincidence of thelandmark is determined. Accordingly, when a plurality of landmarks areextracted, a position error of a landmark with a low degree ofcoincidence can be reduced, and thus the self-position can be estimatedwith high accuracy.

Non-Patent Document 1 discloses a method of estimating a self-positionby associating an image database with an input image. In the estimationmethod, a panorama image to which a position and a direction areassigned in advance is generated for each intersection. Characteristicsextracted from the input image are associated with characteristics of apanorama image group and a vote is cast for a panorama image in whichthe association is possible. The position of a panorama image with themost vote results is determined as the self-position. Next, Homographybetween the input image and the panorama image is calculated, and theinput image is projected on the panorama image using the calculatedHomography, and the projection center is determined as a direction. Inthis case, a position at which the panorama image is captured is used asthe position.

DOCUMENTS OF THE PRIOR ART Patent Documents

[Patent Document 1]

Japanese Unexamined Patent Application, First Publication No. 2008-20225

[Patent Document 2]

Japanese Unexamined Patent Application, First Publication No. 2009-20014

Non-Patent Document

[Non-Patent Document 1]

“Image Based View Localization System Retrieving from a PanoramaDatabase by SURF,” by Naoyuki Yazawa, Hideaki Uchiyama, Hideo Saito,Myriam Servieres, and Guillaume Moreau in 11th IAPR Conference onMachine Vision Applications (MVA), pp. 118-121, May 20-21, 2009.

DISCLOSURE OF INVENTION Problem to be solved by the invention

In Patent Document 1, however, the camera parameter of an image capturedevice is assumed to be known. Therefore, there is a problem thatposition estimation accuracy may deteriorate when a photographer doesnot perform calibration to obtain the camera parameter. Further, inPatent Document 1 and Patent Document 2, it is necessary to position thepositions of the landmarks using the highly accurate GPS or the like.Therefore, a task is required to position the positions or thewidths/heights of the landmarks.

Further, in Non-Patent Document 1, the input image is associated withthe panorama images stored in the database. Therefore, since images arerequired to be captured densely to generate the panorama image, it takessome time to generate the database. Moreover, since the position atwhich the panorama image stored in the database is captured isdetermined as the self-position, there is a problem that theself-position estimation accuracy may deteriorate as the position atwhich the input image is captured is more distant from the position atwhich the panorama image is captured.

An object of the present invention is to provide an image captureposition and image capture direction estimation device, an image capturedevice, an image capture position and image capture direction estimationmethod and a program capable of estimating an image capture position andan image capture direction even when a position (hereinafter referred toas an image capture position; corresponding to the above-describedself-position) at which a photographer photographs (captures) an imageis distant from a position at which an image to which a position and adirection are assigned in advance is captured.

Means for solving the problem

According to the invention, an image capture position and directionestimation device includes: a region determination unit that determinesa plurality of regions to be associated between an image captured by animage capture device and a predetermined image; a depth estimation unitthat estimates depth estimation information corresponding to a depth ofeach of the plurality of regions; and an image capture position andimage capture direction estimation unit that estimates an image capturedirection of the image capture device according to the region with alarge depth and estimates an image capture position of the image capturedevice according to the region with a small depth based on the depthestimation information estimated by the depth estimation unit.

Effects of the Invention

According to the present invention, the region determination unitdetermines the plurality of regions to be associated between the imagecaptured by the image capture device and the predetermined image. Thedepth estimation unit estimates the depth estimation informationcorresponding to the depth of each of the plurality of regions. Theimage capture position and image capture direction estimation unitestimates the image capture direction of the image capture deviceaccording to the region with the large depth and estimates the imagecapture position of the image capture device according to the regionwith the small depth based on the depth estimation information estimatedby the depth estimation unit. Accordingly, for example, by assigning theposition and the direction in advance to the predetermined image, theimage capture position and image capture direction estimation unit canestimate the position and the direction of the image captured by theimage capture device based on the depth estimation information estimatedby the depth estimation unit and the position and the direction assignedin advance to the predetermined image. That is, for example, bypreparing the image to which the position and the direction are assignedirrespective of a specific landmark, it is possible to estimate theposition and the direction of the image captured by the image capturedevice. Accordingly, positioning information regarding a landmark orcamera parameter information may not necessarily be used. Further, thedepth estimation information regarding the region is calculated by thedepth estimation unit. Therefore, by using the depth estimationinformation, it is possible to estimate the image capture position andthe image capture direction, even when the image to which the positionand the direction are assigned in advance is distant from the positionat which the image to which the position and the direction are assignedin advance is captured.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram representing an image capture position andimage capture direction estimation device according to a first exampleof the present invention.

FIG. 2 is a diagram representing examples of a query image to be inputto the image capture position and image capture direction estimationdevice in FIG. 1 and an image to which position direction information isassigned.

FIG. 3 is a block diagram representing an image capture position andimage capture direction estimation device according to a second exampleof the present invention.

FIG. 4 is a flowchart representing an operation of the image captureposition and image capture direction estimation device in FIG. 3.

FIG. 5A is a diagram for describing a specific example of a depth valueof each image capture scene to describe the operation of the imagecapture position and image capture direction estimation device in FIG. 3(an example of an image capture scene: street).

FIG. 5B is a diagram for describing a specific example of a depth valueof each image capture scene to describe the operation of the imagecapture position and image capture direction estimation device in FIG. 3(depth information of FIG. 5A: the larger the depth value is, the whitera color is).

FIG. 5C is a diagram for describing a specific example of a depth valueof each image capture scene to describe the operation of the imagecapture position and image capture direction estimation device in FIG. 3(an example of an image capture scene: indoors).

FIG. 5D is a diagram for describing a specific example of a depth valueof each image capture scene to describe the operation of the imagecapture position and image capture direction estimation device in FIG. 3(depth information of FIG. 5C: the larger the depth value is, the whitera color is).

FIG. 6A is a diagram representing a specific example of a positionestimation contribution distribution according to a depth value.

FIG. 6B is a diagram representing a specific example of a directionestimation contribution distribution according to a depth value.

FIG. 7A is a diagram representing a specific example of a positionmovement amount distribution according to a depth value.

FIG. 7B is a diagram representing a specific example of a directiondeviation amount distribution according to a depth value.

FIG. 8 is a diagram for describing an operation of acquiring a depthvalue at a given characteristic point from depth information of an“outdoor” scene in the image capture position and image capturedirection estimation device in FIG. 3.

FIG. 9 is a block diagram representing an image capture device accordingto another embodiment of the present invention.

EMBODIMENTS FOR CARRYING OUT THE INVENTION First embodiment

Hereinafter, embodiments of the present invention will be described withreference to the drawings. FIG. 1 is a block diagram representing theconstitution of an image capture position and image capture directionestimation device (image capture position direction estimation device)200 according to a first embodiment of the present invention.

The image capture position and image capture direction estimation device200 includes an estimation unit 2. The estimation unit 2 includes aregion determination unit 210 that determines a plurality of regions tobe associated from an image captured by the image capture device, adepth estimation unit 220 that estimates a distant view and a near viewof the extracted region, that is, estimate information (hereinafterreferred to as depth information (depth estimation information))indicating the depth of each region, and an image capture position andimage capture direction estimation unit 230 that estimates an imagecapture position and an image capture direction using the estimateddepth information. The image capture position and image capturedirection estimation device 200 is a device that estimates a positionand a direction of a query image (inquiry image) input from an imageinput device 10 based on the query image. The image capture position andimage capture direction estimation device 200 can be configured toinclude, for example, a CPU (Central Processing Unit) and a storagedevice and can be configured to operate by causing the CPU to execute apredetermined program stored in the storage device.

The query image is, for example, an image that is captured using animage capture device such as a digital camera or a video camera by auser to estimate an image capture position.

The image input device 10 inputs the query image and an image to whichthe position direction information is assigned to the image captureposition and image capture direction estimation device 200. Here, theimage to which the position direction information is assigned is animage to which a position and a direction are assigned in advance. Theimage to which the position direction information is assigned is, forexample, a plurality of still images that are actually captured or animage of each frame of a continuously captured moving image. Theposition and the direction to be assigned are a position of an imagecapture device and an image capture direction at the time of imagecapture. It is not necessary to associate the image to which theposition direction information is assigned according to the presentinvention with a specific landmark and to register the position or thedirection (however, the position and the direction may be registered).Here, an image to which a plurality of pieces of position directioninformation are assigned is input with respect to one query image or aplurality of query images.

The region determination unit 210 determines a plurality of regions tobe used to perform the association between the query image input fromthe image input device 10 and the image to which the position anddirection information is assigned.

The depth estimation unit 220 estimates information indicating a distantview or a near view, that is, a depth, for each of the regionsdetermined by the region determination unit 210 and used to perform theassociation. Further, the estimation result may be expressed by discretevalues of the distant view and the near view or may be expressed bycontinuous values indicating the distant view and the near view. Thatis, the depth estimation unit 220 estimates depth estimation informationcorresponding to a depth of each of the plurality of regions.

Based on the depth information on each of the associated regions, theimage capture position and image capture direction estimation unit 230performs the association on the regions with a large depth (distantview) to estimate the direction from the result and performs theassociation on the regions with a small depth (near view) to estimatethe position from the result. Further, information regarding thedirection and the position estimated by the image capture position andimage capture direction estimation unit 230 is output as image captureposition and image capture direction outputs to a predetermined device.That is, based on the depth estimation information estimated by thedepth estimation unit 220, the image capture position and image capturedirection estimation unit 230 estimates the image capture direction ofthe image capture device according to the regions with the large depthand estimates the image capture position of the image capture deviceaccording to the regions with the small depth. In other words, based onthe depths of the regions associated between the query image captured bythe image capture device and the image to which the position directioninformation is assigned, the image capture position and image capturedirection estimation unit 230 determines whether to estimate theposition or estimate the direction from the associated regions. Then,the image capture position and image capture direction estimation unit230 estimates, as an image capture position, the position of apredetermined image corresponding to a region of which a position isdetermined to be estimated and estimates, as an image capture direction,the direction of a predetermined image corresponding to a region ofwhich a direction is determined to be estimated.

Further, the region with a large depth means a region that has a depthof a relatively large value (that is, a value larger than a depth ofanother region in comparison to the other region) among a plurality ofregions in one image or a region that has a depth of an absolutely largevalue (that is, a value larger than a predetermined reference value).For example, the region with a large depth is a region that has a largedepth among a plurality of regions, a region that has a higher orderthan an order determined in advance when the depth estimationinformation is arranged in a higher order among the plurality ofregions, or a region in which the depth estimation information is largerthan a threshold value determined in advance among the plurality ofregions. Further, like the region with a large depth, a region with asmall depth means a region that has a depth of a relatively small valueamong a plurality of regions in one image or a region that has a depthof an absolutely small value.

Next, an operation according to this embodiment will be described usingexamples of the specific images represented in FIG. 2.

First, the region determination unit 210 determines regions to beassociated from a query image 20 and an image 30 (hereinafter, referredto as an image 30 with position and direction) to which the positiondirection information is assigned. For example, a portion of an imagecorresponding to a mark on a road or a signboard of a shop is set to anassociation region. In the example represented in FIG. 2, 8 regions 21to 28 in the query image 20 and 8 regions 31 to 38 in the image 30 withposition and direction are determined as association regions. In thiscase, the regions 21 to 28 and the regions 31 to 38 are set to beassociated by the image capture position and image capture directionestimation unit 230.

Next, the depth estimation unit 220 determines whether the associationregions 21 to 28 (or the association regions 31 to 38) are distant viewsor near views (that is, estimates the depth of each association region).For example, when the query image 20 is acquired as a moving image and amovement distance of an association region between frames is small, theassociation region is determined as a distant-view region. When amovement distance of an association region between frames is large, theassociation region is determined as a near-view region. That is, forexample, a movement distance between the region 28 of an n^(th) frame(where n is a natural number) of the query image 20 and the region 28 ofan n+1^(th) frame is calculated. When this movement distance between theframes is large, this region is determined as a near-view region.Further, for example, a movement distance between the region 22 of ann^(th) frame of the query image 20 and the region 22 of an n+1^(th)frame is calculated. When this movement distance between the frames issmall, this region is determined as a distant-view region. Further, theestimation of the depth by the depth estimation unit 220 is not limitedto this method. For example, when the query image 20 is a still image,the determination of the distant view and the near view may be performedwith reference to a depth value set for each image capture sceneaccording to a comparison result between a plurality of image capturescenes of near views and distant views registered in advance. Further,the estimation of the depth may be performed by setting the regions 31to 38 or the like in the image 30 with position and direction asreferences.

Next, the image capture position and image capture direction estimationunit 230 associates the regions of the query image 20 and the pluralityof images 30 with position and direction and estimates a direction fromthe associated regions of distant views. For example, the image captureposition and image capture direction estimation unit 230 compares theregions estimated to be the distant views between one query image 20 andthe plurality of images 30 with position and direction, determines theimage 30 with position and direction having a region relatively highlysimilar (or relatively highly homologous) to a region estimated to bethe distant view in the query image 20, and estimates a direction set inthe image 30 with position and direction as a direction of the queryimage 20.

Further, the image capture position and image capture directionestimation unit 230 may perform the association on the plurality ofimages 30 with position and direction, acquire the directions of theassociated images, calculate an average value, and estimate an estimateddirection. Further, the direction may be calculated by multiplying aweight according to the association distance. For example, the directionis valued by increasing a weight as the association distance is shorter.Further, the direction is disvalued by decreasing a weight as theassociation distance is longer. This is because it can be determinedthat the images are captured from substantially the same compositionwhen the association distance is shorter.

Next, the image capture position and image capture direction estimationunit 230 estimates the position in the associated regions of the nearviews. For example, the image capture position and image capturedirection estimation unit 230 compares the regions estimated to be nearviews between one query image 20 and the plurality of images 30 withposition and direction, determines the images 30 with position anddirection having a region relatively highly similar (or relativelyhighly homologous) to a region estimated to be the near view in thequery image 20, and estimates a position set in the image 30 withposition and direction as a position of the query image 20. At thistime, the image capture position and image capture direction estimationunit 230 may correct a position set in the image 30 with position anddirection having a relatively highly similar (or relatively highlyhomologous) region using a depth estimated by the depth estimation unit220, and set the position as the estimated position of the query image20.

Further, the image capture position and image capture directionestimation unit 230 may perform the association on the plurality ofimages 30 with position and direction, acquire positions of theassociated images, calculate an average value, and estimate an estimatedposition. Further, the position may be calculated by multiplying aweight according to the association distance. For example, when theassociation distance becomes shorter, a weight is increased and theposition is regarded to be more important. Further, when the associationdistance becomes longer, a weight is decreased and the position isregarded to be less important. This is because it can be determined thatthe images are captured from substantially the same composition when theassociation distance is shorter.

That is, for example, based on the depths of the regions associatedbetween the query image captured by the image capture device and theimages to which a plurality of pieces of position direction informationare assigned, the image capture position and image capture directionestimation unit 230 determines whether to estimate the position or toestimate the direction from these regions and performs weighting on theimages to which the position direction information is assigned accordingto distances between the associated regions. Then, the image captureposition and image capture direction estimation unit 230 estimates, asthe image capture position, the position of the image to which theposition direction information is assigned and which corresponds to theregion of which the position is determined to be estimated bymultiplying the position by the weight. Further, the image captureposition and image capture direction estimation unit 230 may estimate,as the image capture direction, the direction of the image to which theposition direction information is assigned and which corresponds to theregion of which the direction is determined to be estimated bymultiplying the position by weight.

According to this embodiment, the region determination unit 210determines the plurality of regions to be associated between an imagecaptured by the image capture device and the predetermined image and thedepth estimation unit 220 estimates the depth information according toeach of the depths of the plurality of regions. Then, the image captureposition and image capture direction estimation unit 230 estimates theimage capture direction of the image capture device according to theregion with a large depth based on the depth information estimated bythe depth estimation unit 220, and estimates the image capture positionof the image capture device according to the region with a small depth.At this time, the position and the direction are assigned in advance tothe image 30 with position and direction. Accordingly, the image captureposition and image capture direction estimation unit 230 can estimatethe position and the direction of the query image 20 based on the depthof the region estimated by the depth estimation unit 220 and theposition and the direction assigned in advance to the image 30 withposition and direction. That is, for example, the position and thedirection of the query image 20 can be estimated by preparing the image30 with position and direction to which the position and the directionare assigned. At this time, the information regarding the position andthe direction assigned to the image 30 with position and direction maynot be information corresponding to a specific landmark. That is,landmark positioning information or camera parameter information may notnecessarily be used. Further, the depth estimation unit 220 isconfigured to estimate the depth of the region. Therefore, the imagecapture position can be estimated using the estimated depth information,even when the image 30 with position and direction to which the positionand the direction are assigned in advance is distant from a position atwhich the image is captured.

Second Embodiment

Next, a second embodiment of the present invention will be describedwith reference to FIGS. 3 to 8. FIG. 3 is a block diagram representingthe constitution of an image capture position and image capturedirection estimation device 100 according to the second embodiment ofthe present invention. The image capture position and image capturedirection estimation device 100 according to this embodiment includes anestimation unit 1 and a data storage unit 140. The estimation unit 1includes a characteristic extraction unit 110 that extractscharacteristics from an image and determines regions to be subjected tothe association, a depth estimation unit 120 that estimates depthinformation regarding the regions (hereinafter referred to ascharacteristic regions) from which characteristics are extracted, and animage capture position and image capture direction estimation unit 130that associates the estimated characteristic regions and estimates animage capture position according to the depth information and theassociation result. The image capture position and image capturedirection estimation device 100 can be configured to include, forexample, a CPU and a storage device and can be configured to operate bycausing the CPU to execute a predetermined program stored in the storagedevice. The same reference numerals are given to the same constituentelements as those represented in FIG. 1.

One region or a plurality of regions (for example, characteristic pointsor grids to be described below) from which the characteristics areextracted by the characteristic extraction unit 110 correspond to theassociation regions (for example, the regions 21 to 28 or 31 to 38 inFIG. 2) according to the first embodiment. Further, the characteristicextraction unit 110 is configured to correspond to the regiondetermination unit 210 according to the first embodiment.

The characteristic extraction unit 110 performs predetermined imageprocessing to extract characteristics from a query image input from theimage input device 10 and determines one region or a plurality ofregions from which the characteristics are extracted as regions to besubjected to the association by the image capture position and imagecapture direction estimation unit 130. Further, the characteristicextraction unit 110 extracts characteristics even from an image to whichposition direction information is assigned, and determines one region ora plurality of regions from which the characteristics are extracted. Asa characteristic extraction method, a method of extractingcharacteristics of characteristic points in an image, such as SIFT(Scale-Invariant Feature Transform) characteristics or SURF (Speeded UpRobust Features) characteristics may be used. A method of dividing animage into grids and extracting the characteristics from the grids maybe used.

The depth estimation unit 120 assigns the depth estimation informationstored in a first depth information storage unit 151 to be describedbelow to the characteristic region extracted by the characteristicextraction unit 110. The depth estimation information, that is, a depthvalue dependent on a position in an image, is used when contributiondistribution (position estimation contribution distribution) parametersset to estimate a position by the image capture position and imagecapture direction estimation unit 130 and contribution distribution(direction estimation contribution distribution) parameters set toestimate a direction are read from a second depth information storageunit 152 to be described below.

The image capture position and image capture direction estimation unit130 associates the characteristic regions of the query image with thecharacteristic regions of the image to which the position directioninformation is assigned and calculates amounts of movement between theassociated characteristic regions. The amount of movement means adistance and a direction between the characteristic regions. Further,the image capture position and image capture direction estimation unit130 selects, from the calculated amount of movement between thecharacteristic regions, distributions of a position movement amount anda direction deviation amount for the image to which the positiondirection information is assigned, using the image capture position andimage capture direction estimation amount storage unit 160 that stores amovement amount distribution of the position or the direction. Then, theimage capture position and image capture direction estimation unit 130estimates the image capture position and the image capture directionbased on the position estimation contribution distribution and thedirection estimation contribution distribution read from the seconddepth information storage unit 152 and the position movement amount anddirection deviation amount distributions of all of the characteristicregions.

The data storage unit 140 includes a depth information storage unit 150and an image capture position and image capture direction estimationamount storage unit 160. The depth information storage unit 150 includesthe first depth information storage unit 151 and the second depthinformation storage unit 152. The first depth information storage unit151 stores a depth value according to an image capture scene. The depthvalue may be a relative value by which a magnitude relation of depthscan be understood or may be an absolute value such as a distance value.Further, the second depth information storage unit 152 stores theparameters of the position estimation contribution distribution and theparameters of the direction estimation contribution distribution, andthus such a parameter is selected according to a depth value. Forexample, in the position estimation contribution distribution, thesmaller the depth is, the higher the degree of contribution is. In thedirection estimation contribution distribution, the smaller the depthis, the lower the degree of contribution is. This means that a nearbycharacteristic region is used to estimate the position and a distantcharacteristic region is used to estimate the direction. The imagecapture position and image capture direction estimation amount storageunit 160 stores an amount of movement between the characteristic regionextracted from the query image and the characteristic region extractedfrom the image to which the position and the direction are assigned andthe parameters set to estimate an amount of movement of the position andan amount of deviation of the direction from the depth information.

Next, an operation will be described with reference to the flowchart ofFIG. 4.

First, the characteristic extraction unit 110 inputs a query imagecaptured at a place at which an image capture position and an imagecapture direction are desired to be estimated from the image inputdevice 10 (step S1). Next, the characteristic extraction unit 110extracts characteristics from the image (step S2).

Next, the depth estimation unit 120 assigns the extracted characteristicregions the depth values and the depth information including theparameter of the position estimation contribution distribution andparameter of the direction estimation contribution distribution obtainedfrom the depth value, referring to the depth information storage unit150 (step S3). That is, the depth estimation unit 120 estimates thedepth estimation information by determining an image capture scene ofthe query image and reading the depth information corresponding to thedetermined image capture scene from the first depth information storageunit 151.

Next, the image capture position and image capture direction estimationunit 130 calculates the amount of movement between the associatedcharacteristic regions from the association result between thecharacteristic region of the query image and the characteristic regionof the image to which the position direction information are assigned inadvance, referring to the depth information storage unit 150 and theimage capture position and image capture direction estimation amountstorage unit 160, and estimates the image capture position and the imagecapture direction using the depth information (step S4). Morespecifically, the image capture position and image capture directionestimation unit 130 reads the position estimation contributiondistribution and the direction estimation contribution distribution fromthe second depth information storage unit 152 based on the depthestimation information estimated by the depth estimation unit 120.Subsequently, the image capture position and image capture directionestimation unit 130 reads the position movement amount distribution andthe direction deviation amount distribution from the image captureposition and image capture direction estimation amount storage unit 160based on the corresponding distance and direction between the regionsassociated by the characteristic extraction unit 110. Then, the imagecapture position and image capture direction estimation unit 130estimates the image capture position based on the read positionestimation contribution distribution and the read position movementamount distribution and estimates the image capture direction based onthe read direction estimation contribution distribution and the readdirection deviation amount distribution.

Next, an operation of this embodiment will be described giving aspecific example.

First, data stored in the data storage unit 140 will be described.

The first depth information storage unit 151 stores the depth valueaccording to the image capture scene in advance.

For example, when it is determined that the image capture scene is an“outdoor” scene (see FIG. 5A), the image capture scene has a compositionof a general street and can be defined such that an upper side of theimage is at a long distance from the image capture position and a lowerside of the image is at a near distance extracted from a ground surfaceor the like. In this case, a value obtained when the depth valueincreases from the lower side to the upper side of the image is storedas the depth information (see FIG. 5B). FIG. 5B is a diagramrepresenting an example of the depth information corresponding to theimage of FIG. 5A expressed by grayscale. The whiter a color is thelarger the depth value is.

For example, when it is determined that the image capture scene is an“indoor” scene (see FIG. 5C), the image capture scene can be definedsuch that a center of the image is at a long distance from the imagecapture position and the rest is at a near distance (see FIG. 5D).Further, when various scenes such as an indoor or outdoor scene areassumed as the image capture scenes, the image capture scenes of animage may be determined and the depth information according to the imagecapture scenes may be acquired or a fixed value may be used irrespectiveof the image capture scenes.

The second depth information storage unit 152 stores the parameters ofthe position estimation contribution distribution and the directionestimation contribution distribution dependent on the depth value. Forexample, a depth value x_(a) is defined as a Gauss distributionincluding a dispersion σ_(a) and a coefficient c_(a) as parameters, aposition estimation contribution distribution Npc(x|x_(a, σ) _(a),c_(a)) is expressed by Equation (1), and a direction estimationcontribution distribution Ndc(x|x_(b, σ) _(b), c_(b)) is expressed byEquation (2). For example, in FIG. 6A, the position estimationcontribution distribution Npc has a high value, when the depth value issmall. The position estimation contribution distribution Npc has a lowvalue, when the depth value is large. On the other hand, the directionestimation contribution distribution Ndc in FIG. 6B shows a contrarytendency. This means that a nearby characteristic region is used toestimate the position and a distant characteristic region is used toestimate the direction. FIGS. 6A and 6B are diagrams representing aplurality of examples of the position estimation contributiondistribution Npc and the direction estimation contribution distributionNdc when the horizontal axis represents the depth value and the verticalaxis represents a frequency.

$\begin{matrix}{\; \lbrack {{Equation}\mspace{11mu} 1} \rbrack \mspace{619mu}} & \; \\{{N_{pc}( {{xx_{a}},\sigma_{a},c_{a}} )} = {\frac{c_{a}}{\sqrt{2\; \pi \; \sigma_{a}^{2}}}{\exp ( {- \frac{( {x - x_{a}} )}{2\; \sigma_{a}^{2}}} )}}} & (1) \\{\lbrack {{Equation}\mspace{14mu} 2} \rbrack \mspace{616mu}} & \; \\{{N_{dc}( {{xx_{b}},\sigma_{b},c_{b}} )} = {\frac{c_{b}}{\sqrt{2\; \pi \; \sigma_{b}^{2}}}{\exp ( {- \frac{( {x - x_{b}} )}{2\; \sigma_{b}^{2}}} )}}} & (2)\end{matrix}$

The image capture position and image capture direction estimation amountstorage unit 160 stores the distributions according to an amount ofmovement between the characteristic regions. The amount of movementmeans a distance and a direction. The distance between thecharacteristic regions is calculated from pixel values, and which meterone pixel of the pixel value corresponds to in practice depends on adepth. An amount of movement per pixel may not be uniquely calculated,since camera parameters vary for each kind of camera. Accordingly, adistribution like which meter one pixel corresponds to is generated foreach depth in advance, and parameters of a position movement amountdistribution Npm including an amount of movement and a direction betweenthe characteristic regions are stored (FIG. 7A). For example, when aGauss distribution is assumed, a depth value x_(a) and a distributionNpm(x|x_(a), σ_(a), c_(a)) including a dispersion value σ_(a) and acoefficient c_(a) are set. Likewise, statistical information like whichdegree one pixel corresponds to is generated for each depth, and adirection deviation amount distribution Ndm(x|x_(a), σ_(a), c_(a)) isset (FIG. 7B). FIGS. 7A and 7B are diagrams schematically representingdistributions of an amount of movement and deviation in two kinds ofexamples (the depth values x_(a) and x_(e)) of a position movementamount distribution Npm and a direction deviation amount distributionNdm when the horizontal axis represents the depth value and the verticalaxis represents a frequency.

Next, a case in which the image capture position and image capturedirection estimation device 100 estimates a position from a query imagecaptured by a user will be described.

First, the characteristic extraction unit 110 extracts thecharacteristics from the query image (the query image 20 in FIG. 2) andthe image (the image 30 with position and direction in FIG. 2) to whichthe position direction information is assigned.

Next, when it is determined that the image capture scene of the queryimage is an “outdoor” scene, the depth estimation unit 120 calculates adepth value x_(ai) for each characteristic region from the depthinformation regarding the “outdoor” scene (see FIG. 5A). Further, iindicates an index of the characteristic regions in the image. Further,the position estimation contribution distribution Npc(x|x_(a), σ_(a),c_(a)) and the direction estimation contribution distributionNdc(x|x_(a), σ_(a), c_(a)) are calculated from the depth values of allof the characteristic regions. Here, the depth information is calculatedfor all of the characteristic regions of the query image. However, onlythe associated characteristic regions to be described below may becalculated.

Next, the image capture position and image capture direction estimationunit 130 performs the association between the characteristic pointsextracted by the characteristic extraction unit 110 from the query image(the query image 20 in FIG. 2) and the image (the image 30 with positionand direction in FIG. 2) to which the position direction information isassigned. In the association, distance calculation is performed on eachpoint of the characteristic points of the query image 20 with respect toall of the characteristic points of the image 30 with position anddirection. As the result of the distance calculation, the characteristicpoint of the shortest distance is set as a characteristic pointassociated with the characteristic points of the image to which theposition direction information is assigned. Further, when a differenceof the distance value with the characteristic point of the secondshortest distance in the association is greater than or equal to a givenvalue, the characteristic point may be associated. F_(i)(x, y) isassumed to be an associated characteristic point and F_(i).len isassumed to be a distance value between the characteristic points (seeFIG. 8). FIG. 8 represents an example of a relation between a pluralityof characteristic points indicated by circle marks and a distancebetween the characteristic points indicated by an arrow mark in thequery image indicated by a rectangular frame corresponding to an outdoorscene, as represented in FIG. 5A.

Next, the image capture position and image capture direction estimationunit 130 calculates a position estimation amount and a directionestimation amount. Since a depth zi and a distance value F_(i).lenbetween the characteristic points are determined in a characteristicpoint F_(i), the position movement amount distribution Npm(x|x_(a),σ_(a), c_(a)) in FIG. 7A and the direction deviation amount distributionNdm(x|x_(a), σ_(a), c_(a)) in FIG. 7B can be uniquely selected. Thus, aposition estimation amount F_(pi) at the characteristic point F_(i) iscalculated from a position movement amount distribution N_(pmi) and aposition estimation contribution distribution N_(pci) by Equation (3)below. Further, a direction estimation amount F_(di) at thecharacteristic point F_(i) is calculated from a direction deviationamount distribution N_(dmi) and a direction estimation contributiondistribution N_(dci) by Equation (4) below. Furthermore, with regard toall of the characteristic points, a position estimation distributionF_(p) is calculated using a position estimation amount F_(pi) byEquation (5). Likewise, a direction estimation distribution F_(d) isalso calculated using a direction estimation amount F_(di) by Equation(6). A portion in which the distributions are the maximum is the imagecapture position and image capture direction estimation result. Further,in Equation (3) to (6), X and Π indicate a direct product. Further, σand c, σ_(i), or c_(i) indicate a dispersion and a coefficient of eachdistribution or estimation amount.

$\begin{matrix}{\lbrack {{Equation}\mspace{14mu} 3} \rbrack \mspace{616mu}} & \; \\{{F_{pi}( {{x\sigma_{i}},c_{i}} )} = {N_{pmi} \times N_{pci}}} & (3) \\{\lbrack {{Equation}\mspace{14mu} 4} \rbrack \mspace{616mu}} & \; \\{{F_{di}( {{x\sigma_{i}},c_{i}} )} = {N_{dmi} \times N_{dci}}} & (4) \\{\lbrack {{Equation}\mspace{14mu} 5} \rbrack \mspace{619mu}} & \; \\{{F_{p}( {{x\sigma},c} )} = {\prod\limits_{i}^{N}\; {F_{pi}( {{x\sigma_{i}},c_{i}} )}}} & (5) \\{\lbrack {{Equation}\mspace{14mu} 6} \rbrack \mspace{619mu}} & \; \\{{F_{d}( {{x\sigma},c} )} = {\prod\limits_{i}^{N}\; {F_{di}( {{x\sigma_{i}},c_{i}} )}}} & (6)\end{matrix}$

The image capture position and image capture direction estimation device100 according to this embodiment can estimate the image capture positionand the image capture direction, as in the image capture position andimage capture direction estimation device 200 described above.

The image capture position and image capture direction estimation device100 or 200 according to the above-described embodiment can be applied touse of an image capture position and image capture direction estimationdevice that estimates an image capture position from an image and aprogram that causes a computer to realize the image capture position andimage capture direction estimation device.

As represented in FIG. 9, an image capture device 1000 such as a digitalcamera or a video camera may include the image capture position andimage capture direction estimation device 100 or 200 described withreference to FIGS. 1 to 8. In this case, the image capture device 1000may further include an image capture unit 300, a recording unit 400, anda captured image storage unit 500.

The image capture unit 300 includes an optical system such as a lens oran image capture element such as a CCD (Charge Coupled Device) imagesensor, and thus captures an image of light received via an opticalsystem as a captured image by causing the image capture element toconvert the image into an electric signal. The image capture unit 300also captures the above-described query image.

The captured image captured by the image capture unit 300 is input tothe image input device 10 described with reference to FIG. 1 or 3. Theimage capture position and image capture direction estimation device 100or 200 estimates and outputs the image capture position and the imagecapture direction based on the image input to the image input device 10,as described with reference to FIGS. 1 to 8.

The image capture unit 300 may correspond to the image input device 10described with reference to FIG. 1 or 3. In this case, the image captureposition and image capture direction estimation device 100 or 200estimates and outputs the image capture position and the image capturedirection based on the captured image captured by the image capture unit300, as described with reference to FIGS. 1 to 8.

The recording unit 400 registers and stores the captured image capturedby the image capture unit 300 in the captured image storage unit 500.Further, when the recording unit 400 registers the captured imagecaptured by the image capture unit 300 in the captured image storageunit 500, the recording unit 400 may register and stores the capturedimage in the captured image storage unit 500 in association with theimage capture position and the image capture direction estimated by theimage capture position and image capture direction estimation device 100or 200.

The captured image storage unit 500 is a storage unit that stores thecaptured image and may be a storage medium such as a flash memory.

As described with reference to FIG. 9, when the image capture device1000 includes the image capture position and image capture directionestimation device 100 or 200 according to the above-describedembodiment, a self-position can be estimated by estimating an imagecapture position (and an image capture direction) based on a capturedimage. The self-position refers to an image capture position and animage capture direction at the time of image capture. Therefore, theimage capture device 1000 including the image capture position and imagecapture direction estimation device 100 or 200 according to theembodiment can be applied to use of position positioning under anenvironment in which radio waves of the GPS rarely arrive.

The image capture device 1000 serving as a self-position detectiondevice may include the image capture position and image capturedirection estimation device 100 or 200 and the image capture unit 300.Even in this case, the image capture device 1000 serving as theself-position detection device can estimate the self-position, asdescribed above.

The image input device 10 including the image capture position and imagecapture direction estimation device 100 or 200 described with referenceto FIG. 1 or 3 may be the image capture unit 300 described withreference to FIG. 9. Even in this case, the image capture position andimage capture direction estimation device 100 or 200 serving as theself-position detection device can estimate the self-position, asdescribed above.

The image capture position and image capture direction estimation device100 or 200 according to the embodiment is not limited to theabove-described device. For example, the image capture position andimage capture direction estimation device 100 or 200 and the image inputdevice 10 may be integrally configured or the estimation unit 1 and thedata storage unit 140 may be integrally configured. Further, part or allof a program executed in the image capture position and image capturedirection estimation device 100 or 200 may be distributed through acomputer-readable recording medium or a communication line.

Priority is claimed on Japanese Patent Application No. 2011-069251,filed Mar. 28, 2011, the content of which is incorporated herein byreference.

INDUSTRIAL APPLICABILITY

It is possible to provide an image capture position and image capturedirection estimation device that estimates a position and a direction atthe time of image capture from a captured image without necessity ofpositioning information such as a landmark.

DESCRIPTION OF REFERENCE SYMBOLS

1 Estimation unit

2 Estimation unit

10 Image input device

20 Query image

30 Image with position and direction

100 Image capture position and image capture direction estimation device

110 Characteristic extraction unit

120 Depth estimation unit

130 Image capture position and image capture direction estimation unit

140 Data storage unit

150 Depth information storage unit

151 First depth information storage unit

152 Second depth information storage unit

160 Image capture position and image capture direction estimation amountstorage unit

200. Image capture position and image capture direction estimationdevice

210. Region determination unit

220. Depth estimation unit

230. Image capture position and image capture direction estimation unit

1000. Image capture device

1. An image capture position direction estimation device comprising: aregion determination unit that determines a plurality of regions to beassociated between an image captured by an image capture device and apredetermined image; a depth estimation unit that estimates depthestimation information corresponding to a depth of each of the pluralityof regions; and an image capture position and image capture directionestimation unit that estimates an image capture direction of the imagecapture device according to the region with a large depth and estimatesan image capture position of the image capture device according to theregion with a small depth based on the depth estimation informationestimated by the depth estimation unit.
 2. The image capture positiondirection estimation device according to claim 1, wherein thepredetermined image is an image to which a position and a direction areassigned in advance, and wherein, based on the depth estimationinformation, the image capture position and image capture directionestimation unit estimates the image capture direction according to adirection assigned to the predetermined image corresponding to theregion with the large depth and estimates the image capture positionaccording to a position assigned to the predetermined imagecorresponding to the region with the small depth.
 3. The image captureposition direction estimation device according to claim 1, wherein theimage capture position and image capture direction estimation unitdetermines, based on a depth of the region associated between an imagecaptured by the image capture device and the predetermined image,whether to estimate the position or to estimate a direction from theregion, estimates, as the image capture position, a position of thepredetermined image corresponding to the region of which the position isdetermined to be estimated, and estimates, as the image capturedirection, a direction of the predetermined image corresponding to theregion of which the direction is determined to be estimated.
 4. Theimage capture position direction estimation device according to claim 1,wherein the image capture position and image capture directionestimation unit determines, based on depths of regions associatedbetween an image captured by the image capture device and a plurality ofthe predetermined images, whether to estimate a position or to estimatea direction from the regions and performs weighting on the predeterminedimages according to distances between the associated regions, estimates,as the image capture position, a position of the predetermined imagecorresponding to the region of which the position is determined to beestimated by multiplying the position by the weight, and estimates, asthe image capture direction, a direction of the predetermined imagecorresponding to the region of which the direction is determined to beestimated by multiplying the direction by the weight.
 5. The imagecapture position direction estimation device according to claim 1,further comprising: a first depth information storage unit that storesdepth information corresponding to an image capture scene in advance,wherein the depth estimation unit estimates the depth estimationinformation by determining an image capture scene of an image capturedby the image capture device and reading depth information correspondingto the determined image capture scene from the first depth informationstorage unit.
 6. The image capture position direction estimation deviceaccording to claim 1, wherein the region determination unit extracts acharacteristic from each of an image captured by the image capturedevice and the predetermined image, and determines, as the plurality ofregions to be associated between the image captured by the image capturedevice and the predetermined image, image regions from which thecharacteristic is extracted among image regions of the image captured bythe image capture device and the predetermined image.
 7. The imagecapture position direction estimation device according to claim 6,further comprising: a second depth information storage unit that storesa position estimation contribution distribution and a directionestimation contribution distribution in advance according to the depthestimation information in relation with the depth estimationinformation; and an image capture position direction estimation amountstorage unit that stores a position movement amount distribution and adirection deviation amount distribution according to a distance and adirection between characteristics in relation with the distance and thedirection between characteristics, wherein the image capture positionand direction estimation unit reads the position estimation contributiondistribution and the direction estimation contribution distribution fromthe second depth information storage unit based on the depth estimationinformation estimated by the depth estimation unit, reads the positionmovement amount distribution and the direction deviation amountdistribution from the image capture position direction estimation amountstorage unit based on the distance and the direction betweencharacteristics corresponding to the region associated by the regiondetermination unit, estimates the image capture position based on theread position estimation contribution distribution and the read positionmovement amount distribution, and estimates the image capture directionbased on the read direction estimation contribution distribution and theread direction deviation amount distribution.
 8. An image capture devicecomprising: the image capture position direction estimation deviceaccording to claim
 1. 9. An image capture position direction estimationmethod comprising: determining a plurality of regions to be associatedbetween an image captured by an image capture device and a predeterminedimage; calculating depth estimation information corresponding to a depthof each of the plurality of regions; and estimating an image capturedirection of the image capture device according to the region with alarge depth and estimating an image capture position of the imagecapture device according to the region with a small depth based on thedepth estimation information estimated in the calculating of the depthestimation information.
 10. A program causing a computer to perform:determining a plurality of regions to be associated between an imagecaptured by an image capture device and a predetermined image;calculating depth estimation information corresponding to a depth ofeach of the plurality of regions; and estimating an image capturedirection of the image capture device according to the region with alarge depth and estimating an image capture position of the imagecapture device according to the region with a small depth based on thedepth estimation information estimated in the calculating of the depthestimation information.